Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance

Abstract This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions. Methods such as variable importance measures proposed random forests, partial dependence plots, individual conditional expectation plots remain popular because they are both model-agnostic depend only on pre-trained model output, making them computationally efficient widely available in software. However, numerous studies have found that these tools can produce diagnostics highly misleading, particularly when there is strong among features. The purpose our work here to (i) review this growing body literature, (ii) provide further demonstrations drawbacks along with a detailed explanation why occur, (iii) advocate alternative involve additional modeling. In particular, we describe how breaking dependencies between features hold-out data places undue emphasis sparse regions feature space by forcing original extrapolate where little no data. We explore effects across various setups find support previous claims literature PaP metrics vastly over-emphasize correlated plots. As an alternative, discuss recommend more direct approaches measuring change performance after muting under investigation.